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Abstract 

The  21st  Century  Military  is  shifting  its  focus  from  traditional  combat  operations  to 
stability,  security,  transition,  and  reconstruction  (SSTR)  operations  (Department  of 
Defense  Directive,  2005).  US  policy  on  SSTR  operations  requires  full  interoperability 
among  representatives  across  US  Departments  and  Agencies,  foreign  governments  and 
security  forces,  international  organizations,  US  and  foreign  non-governmental 
organizations,  and  members  of  the  private  sector.  This  shift  in  focus  has  drawn  attention 
to  the  idea  that  one  of  the  most  salient  dimensions  of  21st  Century  warfighting  is  the 
ability  to  operate  effectively  with  others  despite  the  fact  that  there  are  likely  substantial 
differences  in  capabilities  and  cultural  backgrounds.  However,  knowing  the  degree  to 
which  a  team  is  completely  interoperable  can  be  elusive  and  hard  to  quantify.  One 
approach  is  to  evaluate  how  effective  teams  are  able  to  interpret  commander’s  intent  and 
develop  situational  awareness  in  various  conditions.  This  paper  describes  a  technique  for 
evaluating  interoperability  through  a  quantitative  evaluation  of  commander’s  intent  and 
how  intent  was  promulgated  through  a  simulated,  distributed  and  collocated  Coalition 
Task  Force  (CTF)  during  an  October  2006  Defense  Advanced  Projects  Agency  (DARPA) 
sponsored  joint  experiment  between  the  Singapore  (SN)  Armed  Forces  and  the  U.S. 

Army  Research  Laboratory,  Human  Research  and  Engineering  Directorate  (ARL- 
HRED). 


Background 


The  intent  of  this  paper  is  to  leave  the  reader  with  an  appreciation  of  a  SA  (or  Cl) 
technique  using  true/false  probes  that  can  be  used  in  real-time  to  quantify  and  improve 
the  degree  of  interoperability  within  the  context  of  the  challenges  faced  by  Soldiers  in  the 
21st  Century. 

Introduction 

Two  characteristics  define  21st  Century  warfighting;  teamwork  and  the  proliferation  of 
command,  control,  communication,  computers  and  intelligence  (C4i)  systems.  Their 
interaction  form  what  Wame  et  al  (2004)  consider  to  be  one  of  the  most  salient 
dimension  of  21st  Century  warfighting:  the  ability  to  operate  effectively  with  others  in  a 
network  of  C4i  systems.  Salmon  et  al  (2006)  state  that  C4i  systems  comprise  both  human 
and  technological  agents  and  are  designed  to  gather  information  and  facilitate  the 
accurate  communication  of  this  information  between  multiple  agents.  The  expectation  of 
C4i  systems  is  to  allow  a  much  greater  dissemination  and  clearer  exposition  of 
information.  It  is  the  ability  of  the  Commander  to  articulate  his  intent  and  for  the 
command  to  decide  on  the  best  way  of  carrying  that  out.  Alberts  and  Hayes  (2003) 
postulate  that  the  degree  to  which  these  processes,  procedures,  and  C4i  systems  are 
interoperable  will  determine  how  well  collocated  and  distributed  individuals  interpret 
commander’s  intent  and  arrive  at  a  common  situational  awareness. 

The  processes  supporting  an  accurate  awareness  and  interpretation  of  commander’s 
intent  have  never  been  more  complex  and  necessary  than  today.  The  activities  of 
September  11,  2001  and  beyond  marked  a  changing  geopolitical  landscape  where  the 
U.S.  Military  recognized  that  the  Cold  War  modus  of  operandi  is  no  longer  relevant.  A 
decade  ago  America  could  readily  identify  its  adversary  as  the  Soviet  Union,  whose 
stated  intent  was  to  destroy  the  United  States.  However,  the  21st  Century  environment  is 
much  more  complex.  For  example,  the  enemy  is  less  transparent  (i.e.  Disguised  gunmen 
infiltrates  secure  Iraqi  site,  killing  five  Americans  in  Baghdad),  technologically  savvy 
(roadside  bombs),  and  surprisingly  agile.  Evolving  intelligence  shows  that  the  21st 
Century  enemy  is  made  of  complex  and  robust  socio-political  networks  with  national  and 
international  reach-back  capabilities.  With  these  constantly  changing  threats  to  national 
security,  the  requirements  for  U.S.  Commander’s  and  the  military  missions  they  lead 
must  adapt.  To  this  end,  senior  leadership  within  the  U.S.  Department  of  Defense  (DoD) 
has  embarked  on  a  period  of  transformation.  As  part  of  this  transformation,  the  U.S. 
Department  of  Defense  now  recognizes  Stability,  Security,  Transition,  and 
Reconstruction  (SSTR)  operations  as  “core”  military  missions,  giving  SSTR  operations 
the  same  priority  as  traditional  combat  operations  (National  Security  Presidential 
Directive-44,  2005  and  Department  of  Defense  Directive  3000.25,  2006). 


The  expansion  of  military  missions  to  include  SSTR  operations  has  presented  several 
obstacles  to  the  U.S.  military.  While  SSTR  operations  may  involve  force,  these  missions 
are  deeply  connected  to  establishing  and  maintaining  peace.  Therefore,  they  are  not 
warfighting  operations.  SSTR  operations  require  senior  DoD  leadership,  military 
strategists,  and  ultimately  U.S.  Commander’s  to  consider  the  effects  of  their  actions  when 
developing  plans;  a  new  concept  of  operations  titled  Effects  Based  Approach  to 
Operations  (EBAO).  This  new  mindset  for  conducting  military  warfare  also  implies  that 
operational  and  tactical  levels  of  military  planning  consider  the  full  spectrum  of  power; 
political,  economic,  military,  social,  information,  and  intelligence  (PEMSII).  Militaristic 
thinking  and  supporting  information  databases  must  expand  to  include  gathering  and 
interpreting  information  from  a  much  wider  perspective. 

SSTR  operations  also  require  an  unprecedented  amount  of  collaboration  and  cooperation 
across  non-traditional  stakeholders.  At  the  helm  of  these  operations  are  coalition  force 
structures  consisting  of  U.S.  military,  Departments  and  Agencies,  foreign  governments 
and  security  forces,  global  and  regional  international  organizations,  U.S.  and  foreign  non¬ 
governmental  organizations  (NGOs),  and  private  sector  individuals  and  for-profit 
companies,  each  intimately  linked  to  the  other  (Alberts  and  Hayes,  1995).  Paramount  to 
mission  effectiveness  is  the  ability  to  achieve  a  unified  command  team.  Pierce  and 
Bowman  (2002)  noted  that  the  effects  of  cultural  and  organizational  barriers  on  the 
cognitive  fundamentals  of  teamwork,  such  as  coordination  and  decision-making,  must  be 
understood  to  ensure  the  effectiveness  of  CTF.  Findlay  writes  that  participants  were  well 
aware  that  peacekeeping,  stability,  and  reconstruction  operations  (PS&RO)  required  unity 
of  action  from  across  the  intergovernmental  and  international  communities.  For  example, 
the  provincial  reconstruction  teams  (PRTs)  that  have  proven  highly  successful  in 
Afghanistan  are  prime  examples  of  this  approach  to  address  these  complex  contingencies 
(Agoglia,  2005). 

Commander’s  intent  (or  situational  awareness) 

Although  there  has  been  much  progress  in  understanding  teamwork  and  identifying 
characteristics  of  an  effective  team  (see  Paris,  Salas,  &  Cannon-Bowers,  2002  for  a 
review),  developing  quantitative  measures  for  assessing  the  interaction  of  teamwork  and 
C4i  (hereafter  referred  to  as  “interoperability”)  remains  a  challenge  (Salmon  et  al.,  2006 
and  Thomson  &  Adams,  2005).  Recent  methodologies  for  assessing  interoperability 
have  included  discussions  about  situated  cognition  (Miller  and  Shattuck,  2006), 
situational  awareness  (SA;  Gorman  et  al.,  2006  and  Stanton  et  al.,  2006)  and 
Commander’s  Intent  (Farrell,  2004;  Leggatt,  2004;  and  Edgar  et  al.,  2000). 

Situated  cognition,  SA,  and  Cl  all  base  their  approach  on  a  belief  that  a  team  can  only  be 
fully  interoperable  when  each  member  understands  the  purpose  of  the  activity  and  have 
applied  themselves  with  a  unity  of  effort.  While  an  in-depth  discussion  about  each 
approach  is  well  beyond  the  scope  of  this  paper,  the  concepts  and  measures  of  SA  are 
discussed  in  further  detail  as  they  were  directly  applied  in  real-time  to  the  conduct  of  a 
recent  experiment  for  which  ideas  described  in  this  article  are  based.  A  conceptualization 
to  bear  in  mind  when  reading  this  paper  is  that  the  “driver”  of  SA  is  commander’s  intent 


(Cl).  Military  operations  are  never  started  without  some  level  of  understanding  of  the 
situation  and  the  initial  goals  and  actions  to  be  taken.  Leggatt  (2004)  writes  that: 

From  the  beginning  of  the  written  history  of  warfare,  Sun  Tzu  circa  500BC  or 
Alexander  defeating  Darius  in  333BC,  warfighting  has  been  characterized  as  the 
Commander  imposing  his  will  upon  both  his  own  and  enemy  troops  to  achieve  his 
desired  outcome  (Keegan,  1999).  p  2. 

It  is  the  careful  consideration  of  Cl  applied  to  developing  and  dynamic  events  in  the 
environment  about  which  the  Commander  and  his  staff  make  decisions  and  take  action 
(Figure  1).  To  this  end,  SA  and  Cl  may  be  used  interchangeably  throughout  this  paper. 
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Figure  1:  Relationship  between  commander’s  intent  and  situational  awareness,  adapted  from 
Farrell,  2004  p.  4 

A  universally  accepted  definition  and  model  of  SA  does  not  exist  even  after  two  decades 
of  research  and  the  emergence  of  three  dominate  definitions  and  their  accompanying 
theoretical  perspectives  (Salmon  et  al.,  2006).  As  cited  in  Salmon  et  al.  (2006),  the  three 
dominate  theories  of  SA  which  Stanton  et  al.  (2001)  recognize  are  the  three-level  model 
(Endsley,  1995),  the  perceptual  cycle  model  (Smith  and  Hancock,  1995),  and  the  activity 
theory  model  (Bedny  and  Meister,  1999).  The  major  difference  among  the  three 
perspectives  depends  on  whether  SA  refers  to  the  processes  employed  in  achieving  and 
maintaining  it  or  to  the  end  product  derived  as  a  result  of  these  processes.  Therefore,  in 
the  context  of  this  paper,  situational  awareness  is  defined  as  and  refers  to  the  situation- 
specific  information  and  inferences  in  a  person’s  mind,  not  withstanding  those  initially 


shaped  by  the  commander’s  intent,  which  he  or  she  uses  to  make  decisions.  Additionally, 
SA  is  also  believed  to  be  the  combination  of  true  and  false  information,  and  the  premise 
that  good  SA  is  represented  by  the  ability  to  tell  true  and  false  information  apart  (Edgar  et 
al.,  2003). 

Perhaps  more  discouraging  is  that  after  their  literature  review  of  standard  ergonomics 
textbooks,  scientific  journals,  and  existing  human  factors  method  reviews,  Salmon  et  al 
(2006)  identified  only  17,  out  of  30  SA  measurement  techniques  that  were  considered 
appropriate  and  applicable  to  the  study  of  C4i  environments.  Considering  the 
collaborative  and  dispersed  nature  of  C4i  environments,  Salmon  et  al.  (2006)  suggest  that 
measures  of  SA  in  C4i  environments  should  possess  three  capabilities.  C4i  measures 
should  be  able  to  measure  SA  simultaneously  at  different  geographical  locations,  measure 
both  individual  and  team  or  shared  SA,  and  measure  SA  in  real-time. 

The  experimental  environment 

The  experimental  environment  where  the  measurement  of  SA  was  exercised  was  the 
October  2006  Defense  Advanced  Projects  Agency  (DARPA)  sponsored  joint  experiment 
between  the  Singapore  (SN)  Armed  Forces  and  the  U.S.  Army  Research  Laboratory, 
Human  Research  and  Engineering  Directorate  (ARL-HRED).  This  was  an  Operational 
and  Tactical  Level  Command  peace  enforcement  experiment  which  required  participants 
to  role-play  a  Coalition  Task  Force  Headquarters  (CTFHQ)  and  the  Brigade  Tactical 
Command  Posts  as  shown  (Fig  2). 
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Figure  2.  Schematic  Drawing  of  Participant  Distribution  during  Experiment 


The  participants  from  SN  included  students  from  the  Singapore  Command  and  Staff 
College  (SCSC),  operational  officers  from  the  National  Security  Coordinating  Center 
(NSCC),  CBRE  and  G3-Army.  The  participants  from  the  U.S.  included  DARPA 
personnel,  the  Kansas  State  35th  Infantry  Division  (35ID),  the  Joint  Forces  Command 
(JFCOM),  and  the  Edgewood  Chemical  Biological  Center  (ECBC).  In  total,  there  were 
approximately  140  participants  and  data  collectors.  The  objective  of  this  experiment  was 
to  investigate  how  future  C4i  systems  may  alleviate  the  real-world  challenges  faced  by  a 
multinational,  multi-agency  team  in  order  to  facilitate  interoperability. 

The  method  chosen  to  evaluate  SA  was  a  development  of  the  QUASA™  method 
(Quantitative  Analysis  of  Situation  Awareness;  Edgar  et  al.,  2000,  2003;  McGuiness, 
2004  and  Leggatt,  2004).  This  method  uses  probe  statements  about  the  situation  and 
requires  participants  to  judge  whether  they  are  true  or  false.  Their  responses  can  be 
interpreted  using  the  Signal  Detection  Theory  (SDT)  paradigm  which  allows  hits,  misses, 
false  alarms,  and  correct  rejections  to  be  calculated  (Figure  3:  Interpretation  of  Responses 
within  the  Signal  Detection  Theory  Paradigm  below).  This  provides  a  performance 
measure  but  also  gives  an  insight  into  the  bias  of  the  participants.  For  example,  it  is 
possible  to  ascertain  whether  groups  or  individuals  are  more  likely  to  agree  with  false 
information  or  reject  true  information. 
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Figure  3:  Interpretation  of  Responses  within  the  Signal  Detection  Theory 
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There  were  two  experimental  conditions,  1  independent  variable  (interoperability)  with  2 
levels:  non-interoperable  (current  system)  and  interoperable  (future  system).  Within  the 
non-interoperable  condition  (Run  1),  participants  forming  the  SN  and  US  Brigades  and 
the  Coalition  Task  Force  Headquarters  (CTFHQ)  each  used  the  same  SCME-developed 
Command  and  Control  (C2)  software.  For  purposes  of  this  experiment,  C2  software 


capabilities  were  degraded  so  that  Run  1  participants  were  not  supported  by  collaboration 
tools  such  as  video  conferencing,  collaborative  document  editing,  whiteboards,  or  maps. 
Participants  in  Run  1  were  expected  to  communicate  in  face-to-face  meetings  (only 
applicable  for  collocated  teammates)  and  use  basic  text-chat  and  email  systems 
commonly  found  on  today’s  operating  systems.  Run  1  participants  also  shared 
documents  through  use  of  a  shared  hard  drive.  Within  the  interoperable  condition,  Run  2 
participants  used  a  fully  interoperable  mode  of  the  C2  software  provided  in  Run  1 .  This 
allowed  participants  in  Run  2  to  communicate  via  voice-over  internet  protocol  (VOIP), 
video  conferencing,  text  chat  rooms;  as  well  as  use  collaborative  document  editing, 
whiteboards,  and  maps. 

The  dependent  variables  reported  in  this  paper  are  task  performance  to  the  probes  (hit, 
misses,  false  alarms  and  correct  rejections)  which  are  converted  into  d’,  P  and  confidence 
ratings. 

The  hypothesis  developed  was: 

1 .  Hn:  The  participants  in  the  fully  interoperable  condition  in  Run  2  would  have  a  greater 
level  of  SA  sensitivity  than  participants  using  a  degraded  version  of  the  C2  software  in 
provided  in  Run  1 . 

Participants 

116  Soldiers  from  the  Singapore  Armed  Forces,  6  U.S.  Soldiers  from  the  35  Infantry 
Division,  3  U.S.  civilians  from  the  U.S.  Joint  Forces  Command,  1  U.S.  civilian  from  the 
Edgewood  Chemical  and  biological  Center  (ECBC),  and  2  personnel  from  the  Science 
Applications  International  Corporation  (SAIC)  represented: 

1.  Four  different  Brigades  (2  separate  Brigades  for  each  run); 

2.  Two  different  Coalition  Task  Force  Headquarters  (CTFHQ)  (1  CTFHQ  for  each 
run);  and 

3.  One  experimental  control  white  cell  (Same  participants  for  both  runs). 

Data  collection  was  focused  on  49  roles  (CTFHQ  =  25,  Brigades  1  and  2=12  each), 
played  by  94  different  Soldiers.  Due  to  participant  availabilities,  the  same  6  Soldiers 
from  the  35  Infantry  Division  participated  in  both  runs  but  played  different  positions 
between  each  run  (See  Figure  4a  -  b  for  the  breakouts  of  participants  by  organizations 
and  nationality  for  each  run). 
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Figure  4a:  Run  1  participants  by  organization  and  nationality 
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Figure  4b:  Run  2  participants  by  organization  and  nationality 


Their  average  age  was  35  (SD  =  3.5)  and  34  (SD  =  2.3)  for  Run  1  and  2,  respectively. 

The  amounts  of  military  service  were  about  the  same  for  each  run  (Figure  5).  Only  four 
(8%)  of  the  Soldiers  participating  in  run  1  reported  having  fewer  than  10  years  of  military 
service.  Half  (N  =  26,  53%)  reported  between  10  and  15  years  of  military  service  and 
nineteen  (39%)  reported  20  or  more  years  of  military  service. 


Distribution  of  Years  of  Service  by  Run 


Years  of  Service 


Figure  5:  Years  of  Service  by  Run 

However,  the  number  of  participants  reporting  having  had  previous  experience  in 
stability,  security,  transition,  and  reconstruction  (SSTR)  operations  (i.e.  peace  keeping 
(and  peace-  enforcement)  is  vastly  different  by  runs.  More  than  two-thirds  of  Run  1 
participants  (N  =  33,  67%)  indicated  they  have  served  in  SSTR  operations  prior  to  the 
experiment;  fourteen  (28%)  did  not.  The  remaining  4%  (N=2)  had  no  response  or  were 
uncertain.  However,  the  opposite  was  true  of  Run  2  participants,  with  only  18%  (N  =  7) 
reporting  having  had  experience  in  SSTR  operations  prior  to  this  experiment;  thirty 
(79%)  indicated  they  had  no  previous  experience.  These  results  are  summarized  in 
Figure  6  below.  Please  note  that  these  comparisons  are  based  on  a  100%  (N  =  50) 
response  rate  for  Run  1  participants  and  only  76%  (N  =  38)  in  Run  2. 


A  Comparison  of  Stability  Security  Transition  and 
Reconstruction  (SSTR)  Operations  Experience  by 


Experience  in  Peace-Enforcement  Activities 

Figure  6:  Experience  in  SSTR  operations 

There  were  a  total  of  136  participant  roles  to  be  fulfilled  in  the  DARPA  MNDEF  SN-US 
Coalition  C2  experiment.  These  roles  were  divided  into  a  total  of  68  positions  for  each 
run.  25  roles  were  assigned  to  the  Coalition  Task  Force  Headquarters  (CTFHQ),  a  set  of 
12  roles  were  assigned  to  each  of  the  two  Brigades,  and  the  remaining  19  roles  were 
considered  experimental  control  (or  white  cell).  Probe  data  was  only  collected  on 
participant  roles  within  the  CTFHQ  and  both  brigades.  The  data  reported  in  this  report 
include  only  those  participants  who  completed  all  3  sets  of  S A  probes,  2 1  and  23  from 
runs  1  and  2  respectively. 

Procedure 

The  procedure  for  measuring  SA  involved  two  parts,  the  construction  and  the 
administration  of  the  probes.  The  first  set  of  probes  was  developed  directly  from  written 
experimental  documentation  describing  the  experiment  scenario  and  rules  of  engagement. 
The  base  probe  statements  were  later  refined  prior  to  the  start  of  each  experimental  run, 
when  the  white  cell  provided  a  detailed  briefing  about  the  experimental  scenario,  rules  of 
engagement,  and  the  overall  mission.  During  this  period,  the  Commander  of  the  CTFHQ 
and  his  Deputy  were  allowed  to  ask  questions  and  clarify  any  concerns  they  had 
regarding  the  experiment. 

In  addition  to  the  experiment  control  documents  and  pre-experiment  discussions  between 
the  CTFHQ  Commander  and  the  white  cell,  probes  also  were  developed  in  close 
association  to  a  set  of  guidelines  identified  during  the  Multi-national  Experiments  3  and 
4.  Amended  from  the  original  list  of  guidelines  provided  by  Leggatt  (2004,  p.  8)  in  Table 
1,  probe  statements  for  this  experiment  were:  a  direct  reflection  of  commander’s  intent, 
used  simple  language,  a  mixture  of  implicit  and  explicit  issues  to  include  roles  and 
responsibilities,  and  operationally  relevant. 


1 

Reflect  the  commander's  intent 

2 

Use  simple  language  (probes  should  not  be  a  language  test 
for  non-native  English  Speakers 

3 

Not  be  a  verbatim  copy  of  the  published  guidance 

4 

Be  a  mixture  of  implicit  and  explicit  issues 

5 

Be  operationally  relevant 

6 

Be  equal  in  number  true  and  false 

Table  1 :  Guidelines  for  writing  probes 


This  process  for  developing  true/false  probe  statements  was  continued  two  more  times 
throughout  each  experiment  run.  A  total  of  24  probe  statements  were  generated;  3  sets  of 
8  probes.  Each  probe  was  administered  at  the  start,  midpoint,  and  end  of  each 
experimental  run.  The  midpoint  and  ending  of  each  run  conveniently  occurred  around 
naturally  occurring  experimental  breaks  (lunch  and  dinner)  and  therefore  did  not  disrupt 
the  flow  of  the  experiment.  Before  each  break,  experiment  observers  and  analyst 
monitored  the  flow  of  information  and  key  activities  occurring  throughout  each  brigade 
and  the  CTFHQ.  Such  activities  included  Commander’s  side-bar  meetings  with  his 
Deputy  or  Brigade  Commanders.  Data  was  also  collected  during  face-to-face  and  VOIP 
meetings,  and  text-chat.  In  some  cases,  content  for  probes  were  influenced  by  the  direct 
observation  of  the  CTFHQ  Commander’s  text  chat  and  e-mail.  These  correspondences 
served  as  rich  opportunities  for  the  experiment  team  to  continually  update  probe 
statements  between  probe  administrations  to  players.  This  was  one  method  to  ensure 
operational  relevance  of  the  2nd  and  3rd  iterations  of  probes. 

An  example  of  two  probe  statements  is  shown  in  figure  7. 


Sharing  information  with  NGOs  is  in  line  with  the  Commander's  intent  to  avoid 
civilian  casualties.  [T] 

If  the  CTF  receives  media  credit  for  the  recapture  of  the  naval  facility,  this  would 
be  in  line  with  Commander's  intent.  [F] 

Figure  7:  An  example  of  two  probe  statements 

Each  probe  statement  was  also  accompanied  by  a  question  asking  participants  to  provide 
a  confidence  rating  about  their  decision  on  a  scale  from  1  (extremely  unconfident)  to  7 
(extremely  confident). 

Analysis 

The  analysis  of  the  probe  data  was  based  on  the  Signal  Detection  Theory  (SDT).  SDT 
originated  as  a  model  of  perceptual  judgment,  describing  and  analyzing  how  people 
perform  in  tasks  in  which  they  must  detect  a  certain  type  of  stimulus,  a  signal  and  noise. 
The  task  is  to  detect  specific  signals  but  not  confuse  them  with  non-signals  and  other 
irrelevant  stimuli  (noise).  This  task  is  consistent  with  many  real-world  military  decisions, 
especially  SSTR  operations  where  the  Commander  and  his  staff  must  make  sense  of 


incredible  amounts  of  intelligence  from  a  variety  of  sources.  Likewise  was  the  case  in 
this  experiment.  Therefore,  the  probe  data  was  grouped  into  four  categories  (Figure  8): 


“Hit”  -  a  true  statement  which  the  participant  agreed  was  true 
“Miss”  -  a  true  statement  which  the  participant  thought  was  false 
“False  Alarm”  -  a  false  statement  which  the  participant  thought  was  true 
“Correct  Rejection”  -  a  false  statement  which  the  participant  correctly  rejected 

Figure  8:  SDT  Categories 

After  the  data  were  categorized  SDT  statistics  were  applied  -  sensitivity  (d’)  and  bias  (P). 
Sensitivity  is  defined  by  the  SDT  as  an  individual’s  actual  ability  to  discriminate  true 
signals  from  non-signals.  In  the  case  of  true/false  SA  probes,  SA  sensitivity  can  be 
interpreted  as  an  individual’s  ability  to  correctly  discriminate  between  valid  and  invalid 
descriptions  of  the  situation.  The  measure  suggests  that  the  greater  the  sensitivity,  the 
better  the  individual  is  at  making  fewer  misses  and  false  alarms  when  responding  to 
true/false  SA  probes. 

Bias  or  response  criterion  specifies  the  setting  of  an  individual’s  accept/reject  criterion. 
This  can  be  interpreted  as  an  individual’s  leaning  towards  either  more  readily  accepting 
or  more  readily  rejecting  information  when  he  or  she  is  uncertain  as  to  its  validity.  A  low 
P  score  suggests  a  “liberal  or  risk  taking”  strategy  whereas  a  high  p  may  represent  a 
“conservative”  approach.  As  McGuinness  (2004)  notes,  it  remains  to  be  seen  through 
future  research  whether  a  conservative  bias  reflects  a  stable  disposition  of  the  individual 
or  a  more  dynamic  situation-specific  strategy,  or  is  in  fact  peculiar  to  the  probe  technique 
and  is  unrelated  to  SA  itself.  For  this  reason  and  due  to  experiment  limitations  and 
insufficient  data  (multiple  repetitions),  individual  bias  will  not  be  discussed  further. 
Therefore,  the  remaining  paragraphs  of  the  analysis  section,  results,  discussions,  and 
conclusions  are  strictly  based  on  the  sensitivity  statistic  d’  and  confidence  ratings. 

The  data  collected  during  the  experiment  were  aggregated  and  mean  d’  created  for  each 
player,  invoking  the  central  limit  theorem  allowing  the  data  to  be  treated  as  interval  data 
and  analyzed  with  parametric  statistical  tests.  Consistent  with  the  approach  taken  by 
Leggatt  (2004),  on  those  occasions  when  a  participant  failed  to  make  a  false  alarm  and 
thus  making  it  impossible  to  calculate  d’,  the  data  were  modified  in  line  with  Wickens 
(2002).  Wikens  suggests  arbitrarily  modifying  the  FA  rate  from  0  to  0.01  and  likewise 
reducing  the  hit  rate  to  0.99  to  allow  d’  to  be  calculated. 

A  One-way  Analysis  of  Variance  (ANOVA)  design  was  applied  to  test  the  hypothesis 
that  participants  in  Run  2  would  achieve  a  greater  level  of  SA  sensitivity  than  those 
participating  in  Run  1  as  a  function  of  increased  interoperability. 

Confidence  data  was  analyzed  separately  to  the  probe  data.  These  data  were  of  an  ordinal 
nature  but  a  mean  confidence  rating  was  calculated.  These  data  were  also  analyzed  using 
a  One-way  ANOVA. 


Results:  Confidence  Ratings 


Run  differences 


There  were  no  significant  differences  between  runs.  This  suggests  that  within  each  run, 
participants  found  the  probe  statements  to  be  approximately  equal  in  difficulty  to  the 
other.  However,  this  inference  is  not  supported  by  performance  on  SA  probes  which 
suggests  a  significant  difference  between  runs. 

Results:  Sensitivity  (d’) 

Run  differences 


The  ANOVA  identified  a  significant  effect  of  run  F  (1,43)  =  14.858;  P  <  0.01.  This 
finding  may  suggest  that  the  degree  of  interoperability  as  measured  by  the  sheer  amount 
of  available  collaborative  capabilities  does  not  directly  improve  the  likelihood  of  an 
individual’s  ability  to  discriminate  between  valid  and  invalid  descriptions  of  the  situation. 
Therefore,  this  finding  fails  to  reject  the  null  hypothesis. 


Estimated  Marginal  Means  of  Sensitivity  (d’) 


Run 

Run  1:  M:  1.35,  SD:  0.16 
Run  2:  M:  0.49,  SD:  0.10 


Figure  9:  Mean  Sensitivity  (d’)  by  Run 


Discussion 

The  analysis  of  d’  or  sensitivity  identified  that  there  was  a  significant  difference  in 
performance  between  runs.  The  data  suggested  that  participants  in  Run  1  outperformed 
those  participating  in  Run  2  in  their  ability  to  correctly  discriminate  between  valid  and 
invalid  descriptions  of  the  situation.  This  result  was  contrary  to  the  hypothesized 
performance  differences  being  in  favor  of  Run  2  because  of  their  use  of  a  non-degraded, 
fully  interoperable  C2  system. 

There  are  a  number  of  possible  explanations  for  this  finding.  The  high  sensitivity  of  Run 
1  participants  could  suggest  that  the  level  of  interoperability  augmented  through  the 
“degraded”  C2  system  was  sufficient  for  supporting  their  work  processes.  Although  Run 


1  participants  did  not  have  access  to  the  enhanced  collaborative  technologies  assigned  to 
participants  in  Run  2,  they  were  observed  making  extreme  use  of  current  collaborative 
technologies  such  as  text-chat  and  e-mail.  In  addition,  Run  1  participants  were  observed 
spending  a  significant  amount  of  time  in  face-to-face  small  group  meetings. 

The  CTFHQ  Commander  and  his  staff  in  Run  1  also  spent  a  great  deal  of  time  discussing 
strategies  for  monitoring  and  integrating  new  and  evolving  events  into  the  Commander’s 
purview.  Observers  noted  that  one  participant  developed  a  method  for  “flashing” 
important  events  unfolding  about  the  scenario  on  a  large  information  screen  for  all  the 
collocated  CTFHQ  staff  to  view.  Additional  strategies  for  monitoring,  integrating,  and 
disseminating  information  throughout  the  CTFHQ  and  Brigades  emerged  when  the 
political,  information,  diplomatic,  and  media  analyst  agreed  on  a  strategy  for  using  8 
LCD  flat  panel  displays  to  view  relevant  news  reports  and  other  media.  This  information 
became  critical  as  the  CTFHQ  Commander  of  Run  1  re-tasked  a  member  of  his  staff  to 
perform  the  duties  of  Crisis  Manager.  The  Crisis  Manager  became  solely  responsible  for 
ensuring  commander’s  intent  was  understood  throughout  the  CTFHQ  including  the 
Brigades.  This  was  done  through  face-to-face  meetings  with  different  elements  of  the 
command  staff  and  e-mail  correspondence  directly  to  Brigade  Commanders.  The 
creation  of  the  Crisis  Manager  also  proved  to  alleviate  certain  tasks  from  the  Deputy 
Commander  of  the  CTFHQ,  freeing  him  to  coordinate  directly  with  the  CTFHQ 
Commander  and  serve  as  a  liaison  to  the  white  cell. 

Differences  in  leadership  style  between  the  commanders  from  each  run  were  noted.  Run 
1  Commander  was  perceived  as  having  a  stronger  presence  within  the  CTFHQ.  He  was 
also  noted  as  providing  direct  tasking  to  his  Brigades.  The  run  2  Commander  on  the 
other  hand  was  much  more  inclusive  and  involving  of  with  his  staff.  He  pushed  the 
complete  use  of  the  C2  software  and  attempted  to  maximize  its  complete  utility.  Because 
of  this,  Run  2  may  have  become  overwhelmed  or  lost  in  the  C2  tools  capabilities, 
allowing  the  technologies  to  manage  them  versus  them  managing  it.  However,  although 
their  sensitivity  scores  were  lower  than  those  of  Run  1,  several  SME  observing 
experimental  play  were  very  impressed  with  the  staff  within  the  CTFHQ  and  Brigades 
use  of  the  technologies.  It  must  be  noted  that  the  signal  officers  in  Run  2  were  repeatedly 
observed  monitoring  as  many  as  8  different  chats  on  one  display.  These  officers  also 
alerted  the  Commander  in  Run  2  of  emerging  activities  or  changing  initiatives  from  the 
white  cell. 

Conclusion 

This  paper  describes  a  method  and  results  used  by  ARL-HRED  to  quantitatively  measure 
SA  in  a  simulated  CTFHQ  and  two  Brigades  during  the  DARPA  MNDEF  SN-US 
Coalition  C2  experiment  across  two  different  degrees  of  interoperable  conditions.  The 
experimental  context  was  presented  and  its  related  nature  to  the  current  challenges 
regarding  teamwork  and  the  use  of  C4i  systems  was  discussed.  The  use  of  true/false 
probes  to  measure  situational  awareness  was  chosen  because  of  its  ease  of  administration, 
ability  to  be  administered  simultaneously  and  in  real-time,  and  its  minimally  intrusive 
nature.  Previous  research  (Leggatt,  2004)  also  demonstrated  its  applicability  to  assessing 


military  missions  other  than  war  (MOOTW)  such  as  SSTR  operations,  and  its  ability  to 
capture  or  suggests  differences  in  teamwork  and  the  usage  of  C4i  systems  for  information 
gathering  and  synthesis. 

The  continual  challenge  of  research  using  probe  statements  is  the  actual  probes 
themselves.  The  language  and  composition  of  each  probe  may  always  be  reworded  for 
improved  understanding.  More  time  can  always  be  spent  examining  the  salient  aspects  of 
commander’s  intent  before  developing  each  probe  statement.  For  simplicity  sake,  the 
probes  in  this  experiment  were  developed  using  four  questions  we  believed  summarized 
the  essence  of  situational  awareness  and  its  interaction  with  commander’s  intent:  What  is 
happening?  Why  is  it  happening?  What  does  it  mean  in  terms  of  my  objectives  defined 
by  the  Commander?  What  actions  have  I  been  directed  to  take  as  a  result  of  the 
developing  situation  and  the  information  I  have  gathered. 


Although  we  are  continually  adapting  to  both  the  inclusion  of  non-traditional  military 
missions  such  as  peacekeeping  and  peace  enforcement  and  the  use  of  C4i  technologies, 
the  findings  from  this  report  cautiously  highlight  the  importance  that  additional 
technological  capabilities  are  not  solutions  in  and  of  themselves.  Future  research  must 
continue  to  replicate  real-world  environments,  and  the  measurement  techniques  to  follow 
must  be  reliable,  applicable,  simple,  and  robust  enough  to  capture  any  differences  in 
performance  that  may  exists.  Regardless  of  the  amount  of  technology  that  is  being 
provided  in  the  field,  and  whether  researchers  agree  on  definitions  of  situational 
awareness  or  commander’s  intent,  the  Commander  and  his  staff  will  always  need  to  be 
able  to  answer  the  questions:  do  we  know  what  we  really  know  and  what  we  do  not 
know.  Answers  to  these  questions  can  only  be  explored  through  careful  experimental 
designs  and  applicable  measurement  techniques. 
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•  The  challenge  for  research,  development, 
technology,  and  engineering  (RDT&E) 

•  Commander’s  intent  (Cl),  Situational  Awareness 
(SA),  Situational  Understanding  (SU) 

•  Problem  statement  -  measuring  Cl  /  SA 

•  True  /  False  Probes 

•  T/F  Probes  applied  in  experiment 

•  Conclusions  &  discussion 
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Communications  Services  I  Command  Operating  Procedures 


RDT&E  Challenge 


Human  Research  &  Engineering  Directorate 


Designing  C4ISR  technologies  and  organizational  networks  with  human 

behavior  in  mind 
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Today 


Military  Decision 
Making  Process 
using  an  Effects 
Based  approach 
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Outcomes 


Challenge:  continuously  anticipate  &  adapt  to  an  emergent  situation 


Human  Research  &  Engineering  Directorate 


How  are  they  related? 


s  Process  oriented 


S  Knowledge  acquisition 

S  Technology  enabled 

Cl  is  a  statement  of  what  the  force 
must  do  and  the  conditions  the 
force  must  meet  to  succeed 


SA  is  knowledge  of  the  present 
environment 

SU  is  understanding  relationships 
about  factors  in  the  environment 


Cl  /  SA  /  SU 


Source:  The  Operations  Process,  FMI  5-0.1,  March  2006 
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Measuring  Cl  /  SA  /  SU 


Human  Research  &  Engineering  Directorate 


ARL’s  approach  to  measuring  SU  was 
recognizing  that  the  process  for 
developing  SU  required  that  individuals 
collect  and  process  data  and  make 
inferences  about  that  data  bounded  by 
an  understanding  of  the  commander’s 
intent  and  knowledge  about  the  roles 
and  responsibilities  assigned  to  them. 


Developing  SU,  FMI  5-0.1 
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Origin 


•  True  /  False  Probe 
Technique  and  SDT  Analyses 

Edgar,  G.K.,  Smith,  A.J., 
Stone,  H.E.,  Beetham,  D.L., 
Pritchard,  C„  (2000). 

QUASA:  Quantifying  and 
Analyzing  Situational 
Awareness.  Paper 
presented  at  the  IMCD 
People  in  Digitized 
Command  and  Control 
Symposium,  RMCS 
Shrivenham,  UK  (CD-ROM 

•  T/F  Probe  and  SDT 
applied  to  Commander’s 
Intent 

Multi-National 
Experiments  3  and  4 
(Leggatt,  2005) 


T/F  Probe  Methodology 


Human  Research  &  Engineering  Directorate 


Guidelines 


•  Reflect  Commander’s 
Intent 

•  Be  a  mixture  of  implicit 
and  explicit  issues 

•  Use  simple  language 
(probes  should  not  be  a 
language  test  for  non¬ 
native  English  speakers) 

•  Not  be  a  verbatim  copy 
of  published  guidance 

•  Operationally  relevant 

Source:  (Leggatt,  2005) 


Sample  Probes 


•  The  Commander  wants  the 
positive  activities  of  the 
military  to  receive  media 
attention  (T) 

•  The  Commander’s  main 
objective  is  to  capture  the 
territory  of  Jurongland  (F) 


•  The  Commander’s  intends 
to  take  every  effort  in  IO 
before  commencing  air- 
strikes  in  Middleland  (T) 
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Experiment 

Objectives 

Develop  and  evaluate  technologies  to 
improve  cognitive  interoperability  & 
collaboration  among  joint,  interagency, 
&  multinational  forces  conducting 
combat  &  stability  operations. 


.  conditions  Methods 

-  Run  1:  Degraded  System  Interoperability  or 
collaboration  tools 

-  Run  2:  System  Interoperability  +  advanced 
collaboration  tools  &  team  performance 
feedback 

•  Players 

-  130  Singapore  Armed  Forces  (SAF)  &  Staff 
College  Students 

-  US  military  officers  (35ID&JFCOM) 

-  Representatives  of  US,  SN,  other  government 
agencies  and  non-government  organizations 

•  Location 

SAF  Centre  for  Military  Experimentation,  Singapore, 
October  2-14  2006 


US  -  Singapore 
in  Coalition  Operations 


Measures 


•  Situational 

-  Communication  Analysis 

-  Leader  and  Team  Trust 

-  Team  Experiences 

-  Workload 

•  Dispositional 

-  Cultural  Understanding  - 
GlobeSmart  Commander 

-  Uncertainty  Response  Scale 

-  Conflict  Avoidance 

•  Outcome  Measures 

-  Situation  Understanding 

-  Commitment 


Experiment  Conditions 


Human  Research  &  Engineering  Directorate 


Trial 

Conditions 

Emulated  using 
private  services 
for  voice,  chat, 
message 

Run  1:  Each  entity 
employs  its  own 
computing  and 
communications 
capability;  not 
interoperable. 

■  oim  uses  current  oin  oo  capamimes  ^not  luenuneu; 

■  US  Military  uses  current  SN  C3  capabilities 

■  US  non  Military  uses  private  services  (voice,  chat,  message) 

■  Other  entities  use  public  services  (voice,  chat,  message) 

Run  2:  Each  entity 
employs  a  fully 
interoperable  computing 
and  communications 
capability. 


■  All  entities  use  CATISA  (SN  software  collaboration  tools)  in  full 
collaborative  mode  to  represent  an  interoperable  collaboration 
system 

■  Selected  entities  will  receive  TeamViz  feedback  about  team 
communication  and  collaboration 
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[igher  Authorities 


Singapore 


2  US  (JFCOM) 


Multiple  Trials 


Run  2 


Run  1 


22  Singapore 


BDE/BN  TOC 
12  Singapore 
(6  +  3  *2) 


Coy  Comd 
7  Singapore 


2  x  Media 


Red  Cell 
5  x  Singapore 


Experiment  Organization 


Singapore 

Gov’t 
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Gov’t 


US  Gov’t 


UN 


Singapore 

Embassy 


US  Embassy 


SG  CMD 
and  Staff 


White:  Ceil 


Coalition  Task  Force  HQ 


US  CMD 
and  Staff 


4  US  (National  Guards) 
2  x  Pol-Mil,  2  x  Analyst 


Other  Government  Agencies:  SN 
Ministry  of  Foreign  Affairs,  U.S. 
State  Dept,  and  CDC 


OGA  Cell 
1  Singapore 
1  US 


BDE/BN  TOC 
8  Singapore 
(6+2) 


Adversary 
Coy  Comd 


BDE/BN  TOC 

4  US  players  (National  Guards  = 
2*2  US  BN) 

Bde2IC/Bn  CO,  Bde  DyS3/Bn 
CO,  2  x  Bn  S3/Bde  AS3 


Observers 
TBD  Singapore 
3  US 
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Human  Research  &  Engineering  Directorate 


Experiment  Hypothesis 


Future  systems  (collaborative  and  context  aware)  will 
improve  efficiency  and  effectiveness  of  coalition  civil- 
military  operations,  as  compared  to  current  systems 
(Run  1 ),  in  terms  of: 

-  Degree  of  understanding  the  situation 

-  Degree  of  understanding  mission  and  “commander’s  intent” 

-  Ability  to  adapt  to  unforeseen  critical  incidents 

-  Plan  confidence 

-  Level  of  trust  between  team  members 

-  Communication  patterns  and  content 

•  more  likely  to  communication  within/between  teams 

•  awareness  of  similar  discussions  that  occur 

•  awareness  of  groupthink 
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Results:  Situational  Understanding 


Human  Research  &  Engineering  Directorate 


Objective:  To  determine  if  a 
difference  in  performance  exists 
between  a  non-collaborative 
(Run  1)  and  a  fully  collaborative 
(Run  2)  distributed  and 
collocated  multinational 
CTFHQ? 


End  of  Experiment 

Probability  of  Making  a  Correct  Response,  Runs 
1  and  2:  Plot  of  Means 


Run  1: 

N:  23  (Represents  only 
43%  of  data  due  to 
incomplete  responses) 

M:  0.7717  (±0.028) 

Run  2: 

N:  21  (Represents  only 
53%  of  data  due  to 
incomplete  responses) 

M:  0.6995  (±  0.029) 


Extremely 

confident 

▲ 


▼ 


Extremely 

unconfident 


Run  1  -  Team  Confidence  Ratings: 


Run  2  -  Team  Confidence  Rating: 


Plot  of  Means 


■  Bde  1 

■  Bde  2 

■  CTFHQ 


Probe  1 


Probe  2 


Probe  3 


Probe  1 


Probe  2 


Probe  3 
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Results:  Situational  Understanding 


Human  Research  &  Engineering  Directorate 


Bottom  Line  Up  Front:  Run  1  participants  out  performed  Run  2 


Situational  Understanding:  A 
significant  difference  exists  in  a 
CTFHQ’s  ability  to  identify  true 
statements  about  the  situation 

F  (1,43)  =  14.858,  p  =  . 000433 
Runl:  N  =  23,  M:  1.35,  SD:  0.159 
Run  2:  N  =  21,  M:  0.49,  SD:  0.102 


Estimated  Marginal  Means  of  Situational 
Understanding 
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Human  Research  &  Engineering  Directorate 


Conclusion 


*  Confidence  Ratings 

-  Participants  are  generally  confident  about  their  responses 

-  Appears  to  be  a  difference  between  the  CTF  and  Brigades  in  Run  1 , 
while  Brigade  2  in  Run  1  were  less  confident  than  the  CTF  and  Brigade  1 

-  Anecdotal:  The  closer  you  are  to  the  CTFHQ,  the  more  confident  you  are 
in  your  ability  to  gain  situational  understanding 

*  Correct  Response 

-  Overall,  Run  1  CTFHQ  demonstrated  the  highest  probability  of  making  a 
correct  response  with  its  CTF  earning  the  highest  probability  of  correct 

responses  (0.8071  ±  0.021) 

*  Situational  Understanding 

-  Run  1  CTFHQ  performed  better  than  Run  2  CTFHQ  at  discriminating  true 
versus  false  statements 

-  Anecdotal:  SU  was  probably  greater  in  Run  1  CTFHQ  due  to  an  overall 
confidence  and  understanding  of  the  traditional  planning  process,  and  the 
result  of  emerging  strategies  for  improved  information  management 
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T  /  F  Probes 


1st  Iteration 

1 .  The  CTF  Commander  wants  the  positive  activities  of  the  military  to  receive  media  attention.  [T] 

2.  If  the  opposing  force  were  to  expand  their  operations  beyond  the  Naval  port,  The  Freedom  CP  has  full  and  immediate  authority  to 
engage  the  enemy.  [T] 

3.  The  CTF  Commander  notes  that  Terrorist  suspects  are  not  a  part  of  the  Middleland  forces.  [T] 

4.  The  role  of  the  Intel  Analysts  is  to  assist  in  finding  out  potential  suspects  seeking  to  increase  violence  in  Jurongland.  [T] 

5.  The  CTF  Commander  notes  that  military  intervention  in  the  area  must  be  persistent  and  consistent.  [T] 

6.  Jurongland  has  been  investing  in  chemical  and  biological  agents.  [F] 

7.  The  CTF  Commander  does  not  regard  the  establishment  of  basic  human  rights  as  a  priority.  [F] 

8.  The  CTF  Commander's  main  objective  is  to  capture  the  territory  of  Jurongland.  [F] 

2nd  Iteration 

1 .  The  establishment  of  the  No-Fly  Zone  and  Zone  of  Separation  in  Middleland  is  to  not  disrupt  civil  or  economic  activities.  [T] 

2.  The  Commander  intends  to  use  international  pressure  and  economic  sanctions  to  deter  Middleland  from  further  invasion.  [T] 

3.  The  Commander  intends  to  use  a  show  of  force  if  evacuations  create  additional  violence. [T] 

4.  The  Commander  regards  the  establishment  of  security  and  peace  as  a  priority  over  and  above  recapturing  the  Naval  facility.  [T] 

5.  The  Commander  intends  to  take  every  effort  in  IO  before  commencing  air  strikes  in  Middleland.  [T] 

6.  The  Tiger  Beer  truck  explosion  was  attributed  to  the  Middleland  Forces.  [F] 

7.  ECA  B  (Economic  Survival)  is  deemed  to  be  the  most  dangerous  for  the  coalition  forces. [F] 

8.  1st  Bde  is  to  assist  the  Jurongland  Homeland  Security  Forces  in  maintaining  peace  and  security.  [F] 

3rd  Iteration 

1 .  The  establishment  of  the  No-Fly  Zone  and  Zone  of  Separation  in  Middleland  is  to  not  disrupt  civil  or  economic  activities.  [T] 

2.  The  Commander  intends  to  use  international  pressure  and  economic  sanctions  to  deter  Middleland  from  further  invasion.  [T] 

3.  The  Commander  intends  to  use  a  show  of  force  if  evacuations  create  additional  violence. [T] 

4.  Sharing  information  with  the  NGOs  is  in  line  with  the  Commander's  intent  to  avoid  civilian  casualties.  [T] 

5.  If  the  CTF  receives  media  credit  for  the  recapture  of  the  naval  facility,  this  would  be  in  line  with  Commander's  intent.  [F] 

6.  The  Commander  does  not  regard  the  establishment  of  a  strong  Jurongland  government  as  a  priority.  [F] 

7.  ECA  B  (Economic  Survival)  is  deemed  to  be  the  most  dangerous  for  the  coalition  forces. [F] 

8.  2nd  Bde  is  tasked  to  defend  the  Naval  facility.  [F] 
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